<html>
<head><meta charset="utf-8"><title>expanding procedural macros · t-compiler/rust-analyzer · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/index.html">t-compiler/rust-analyzer</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/expanding.20procedural.20macros.html">expanding procedural macros</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="166402247"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/expanding%20procedural%20macros/near/166402247" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/expanding.20procedural.20macros.html#166402247">(May 23 2019 at 21:34)</a>:</h4>
<p>I've just read a master's thesis on implementing procedural macros in IntelliJ Rust, and I think we could reuse this approach (and even code!)</p>
<p>The key idea is to isolate bits that use unstable compiler's API into a separate process, and write a stable IPC shim on top. This not only  doges stability problems, but also greatly increases reliability: run away proc macro won't be able to crash the IDE process.</p>
<p>The shim is here: <a href="https://github.com/fedochet/rust-proc-macro-expander" target="_blank" title="https://github.com/fedochet/rust-proc-macro-expander">https://github.com/fedochet/rust-proc-macro-expander</a> I believe we can basically just use it?</p>



<a name="166402285"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/expanding%20procedural%20macros/near/166402285" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/expanding.20procedural.20macros.html#166402285">(May 23 2019 at 21:35)</a>:</h4>
<p>Also, kudos to <a href="https://github.com/fedochet" target="_blank" title="https://github.com/fedochet">https://github.com/fedochet</a>, masters student at <a href="https://en.itmo.ru/" target="_blank" title="https://en.itmo.ru/">https://en.itmo.ru/</a> who did all the work!</p>



<a name="166402403"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/expanding%20procedural%20macros/near/166402403" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/expanding.20procedural.20macros.html#166402403">(May 23 2019 at 21:36)</a>:</h4>
<p>cc <span class="user-mention" data-user-id="216201">@Edwin Cheng</span></p>



<a name="166402465"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/expanding%20procedural%20macros/near/166402465" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/expanding.20procedural.20macros.html#166402465">(May 23 2019 at 21:37)</a>:</h4>
<p>also cc <span class="user-mention" data-user-id="153740">@Igor Matuszewski</span>, the overall architecture  pattern, when rustc_private bits are moved into a separate process looks like something we want for both RLSes</p>



<a name="166416713"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/expanding%20procedural%20macros/near/166416713" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/expanding.20procedural.20macros.html#166416713">(May 24 2019 at 02:01)</a>:</h4>
<p>That's neat !</p>



<a name="166421678"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/expanding%20procedural%20macros/near/166421678" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/expanding.20procedural.20macros.html#166421678">(May 24 2019 at 04:14)</a>:</h4>
<p>I did dig this problem a litte bit recently, the real hard part of the problem is , there is no way to call a proc-macro directly.<br>
For example, imagine following proc-macro:</p>
<div class="codehilite"><pre><span></span><span class="cp">#[proc_macro_attribute]</span><span class="w"></span>
<span class="k">pub</span><span class="w"> </span><span class="k">fn</span> <span class="nf">foo</span><span class="p">(</span><span class="n">attr</span>: <span class="nc">TokenStream</span><span class="p">,</span><span class="w"> </span><span class="n">item</span>: <span class="nc">TokenStream</span><span class="p">)</span><span class="w"> </span>-&gt; <span class="nc">TokenStream</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="c1">// ...</span>
<span class="p">}</span><span class="w"></span>
</pre></div>


<p>Instead of the normal usage :</p>
<div class="codehilite"><pre><span></span><span class="cp">#[mycrate::foo]</span><span class="w"></span>
<span class="k">fn</span> <span class="nf">wrapped_function</span><span class="p">()</span><span class="w"> </span><span class="p">{}</span><span class="w"></span>
</pre></div>


<p>We want to call it like this:</p>
<div class="codehilite"><pre><span></span><span class="k">fn</span> <span class="nf">some_func</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w">    </span><span class="kd">let</span><span class="w"> </span><span class="n">token_stream</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">mycrate</span>::<span class="n">foo</span><span class="p">(...);</span><span class="w"></span>
<span class="w">    </span><span class="c1">// ...</span>
<span class="p">}</span><span class="w"></span>
</pre></div>


<p>However, in the current stable rustc there is no way to call a proc-macro directly.<br>
There are some solutions and idea about it:<br>
1. <strong>(UNSTABLE)</strong> Same as the link you provided, loading the <code>proc-macro</code> crate dylib manually, and use unstable api (i.e. <code>proc_macro::bridge</code>) to bypass it.<br>
2. Add a compiler flag in rustc to bypass the <code>proc_macro_attribute</code> and change the crate type to <code>rlib</code>.<br>
3. Change rustc proc-macro to use IPC and deal with that IPC manually.<br>
4. <strong>(STABLE but complicated)</strong> Write a "copy-on-write" virtual file system (FUSE?) and overwrite the given proc-crate <code>Cargo.toml</code> and its source code, to mask all proc-macro related stuffs and compile it as a normal <code>rlib</code> crate.</p>



<a name="166421683"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/expanding%20procedural%20macros/near/166421683" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/expanding.20procedural.20macros.html#166421683">(May 24 2019 at 04:14)</a>:</h4>
<p>Personally, I want to try to implement <strong>4</strong>, and happy to use (but not implement) <strong>2</strong>. But for fast prototyping, <strong>1</strong> is okay.</p>



<a name="166422138"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/expanding%20procedural%20macros/near/166422138" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Edwin Cheng <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/expanding.20procedural.20macros.html#166422138">(May 24 2019 at 04:26)</a>:</h4>
<p>It is because seem like <code>proc_macro::bridge</code> are far from stabilized. Although this part of "proc-macro" slim  is out of the scope of the main <code>rust-analyzer</code> project and it just use the nightly compiler <strong>indirectly</strong>,  depends on a nightly toolchain is not ideal for me.</p>



<a name="166433627"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/185405-t-compiler/rust-analyzer/topic/expanding%20procedural%20macros/near/166433627" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> matklad <a href="https://rust-lang.github.io/zulip_archive/stream/185405-t-compiler/rust-analyzer/topic/expanding.20procedural.20macros.html#166433627">(May 24 2019 at 08:40)</a>:</h4>
<p>Approach 4 sounds interesting! I am afraid though, that it'll have a lot of overhead. For example, it would be hard to batch expansinos of several macros. In a sense, approach 4 already exists as a <code>cargo expand</code> subcommand. </p>
<p>Approach 1 seems fine to me:</p>
<ul>
<li>as the shim's scope is limited, it's OK to depend on nightly compiler</li>
<li>we want a separate process for this long-term anyway, so this approach actually is the first step in the long-term direction</li>
</ul>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>